Image Processing and Segmentation of Sets of Z-Stacked Images of Three-Dimensional Biological Samples

ABSTRACT

Methods are provided to project depth-spanning stacks of limited depth-of-field images of a sample into a single image of the sample that can provide in-focus image information about three-dimensional contents of the image. These methods include applying filters to the stacks of images in order to identify pixels within each image that have been captured in focus. These in-focus pixels are then combined to provide the single image of the sample. Filtering of such image stacks can also allow for the determination of depth maps or other geometric information about contents of the sample. Such depth information can also be used to inform segmentation of images of the sample, e.g., by further dividing identified regions that correspond to the contents of the sample at multiple different depths.

BACKGROUND

A variety of biological experiments include the analysis of a great manysamples, each of which may be associated with a number of parameters orother information generated via measurement or assessment of the sample.Such samples may include cells or other biological contents, each samplediffering with respect to growth medium (e.g., hormones, cytokines,pharmaceuticals, or other substances in the growth medium), source(e.g., cultured, biopsied or otherwise explanted from natural tissue),incubation conditions (e.g., temperature, pH, light level or spectrum,ionizing radiation), or some other controlled conditions in order toobserve the response of the cells or other biological contents to theapplied conditions. This could be done, e.g., in order to assess theresponse of the samples to a putative therapy, to elucidate somebiological process, or to investigate some other question of interest.

Assessing such samples may include using a microscope, fluorescenceimager, or other means to microscopically image the samples. Inpractice, the imaging of samples across a large depth of field may beimpeded by limitations of the optics or other apparatus used to imagethe samples. For example, an objective or other elements used to imagethe sample may be limited with respect to depth of focus, making itdifficult or impossible to simultaneously image, in focus, the entiretyof a sample that spans a volume that is larger, along the optical axisof a device used to image the sample, than the depth of focus of theimaging apparatus. Such “three-dimensional” samples, as contrasted withsamples that are spread across a glass slide and/or that have beensectioned into slices such that they span a volume that can fit withinthe depth of field of an imaging apparatus, may contain a variety ofstructures of interest, e.g., organoids, tumor spheroids, or otherthree-dimensional multicellular structures.

SUMMARY

An aspect of the present disclosure relates to a method for generating aprojection image of a three-dimensional sample, the method including:(i) obtaining a set of images of the sample, wherein each image of theset of images corresponds to a respective focal plane within the sample;(ii) applying a filter to each image of the set of images to determine arespective depth value for each pixel of an output image of the sample,wherein a given depth value represents a depth, within the sample, atwhich the contents of the sample can be imaged in-focus; and (iii)determining an image value for each pixel of the output image based onthe depth value of the pixel of the output image. Determining an imagevalue for a particular pixel of the output image includes: (1)identifying an image of the set of images that corresponds to the depthvalue of the particular pixel; and (2) determining the image value forthe particular pixel based on a pixel, of the identified image, having alocation within the identified image that corresponds to the particularpixel.

Another aspect of the present disclosure relates to a method forgenerating a projection image of a three-dimensional sample, the methodincluding: (i) obtaining a set of images of the sample, wherein eachimage of the set of images corresponds to a respective focal planewithin the sample; (ii) applying a filter to each image of the set ofimages to determine a respective depth value for each pixel of a depthmap, wherein the depth value represents a depth, within the sample, atwhich contents of the sample can be imaged in-focus; and (iii)determining an image value for each pixel of an output image based onthe depth value of a corresponding pixel of the depth map.

Yet another aspect of the present disclosure relates to a method forsegmenting an image of a sample, the method including: (i) obtaining animage of the sample; (ii) obtaining a depth map of contents of thesample; (iii) generating a first segmentation map of the sample based onthe image; and (iv) based on the depth map, generating a secondsegmentation map of the sample by further dividing at least one regionof the first segmentation map.

Yet another aspect of the present disclosure relates to acomputer-readable medium that is configured to store at leastcomputer-readable instructions that, when executed by one or moreprocessors of a computing device, cause the computing device to performcomputer operations carrying out one or more of the methods describedherein. Such a computer-readable medium could be a non-transitorycomputer-readable medium.

Yet another aspect of the present disclosure relates to a systemincluding: (i) one or more processors; and (ii) a non-transitorycomputer-readable medium that is configured to store at leastcomputer-readable instructions that, when executed by the one or moreprocessors, cause the system to perform one or more of the methodsdescribed herein.

These as well as other aspects, advantages, and alternatives will becomeapparent to those of ordinary skill in the art by reading the followingdetailed description with reference where appropriate to theaccompanying drawings. Further, it should be understood that thedescription provided in this summary section and elsewhere in thisdocument is intended to illustrate the claimed subject matter by way ofexample and not by way of limitation.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a cross-sectional schematic view of a sample being imaged byan imaging system.

FIG. 2A depicts a set of example images of a sample.

FIG. 2B depicts a projection image generated based on the set of exampleimages of FIG. 2A.

FIG. 3A depicts an example projection image and an example depth map ofa sample.

FIG. 3B depicts set of example images of the sample used to generate thedepth map of FIG. 3A.

FIG. 3C is an example set of texture values determined across a range ofdepths for a sample.

FIG. 4 depicts set of example images of a sample used to generate aprojection image of the sample.

FIG. 5A is an example image of a sample.

FIG. 5B is an example first segmentation map of the sample representedby the image of FIG. 5A.

FIG. 5C illustrates particular segments of the first segmentation map ofFIG. 5B, overlaid onto an example depth map.

FIG. 5D is an example second segmentation map determined from the firstsegmentation map of FIG. 5B.

FIG. 6 illustrates in perspective view elements of an example system.

FIG. 7 is a schematic illustration of elements of an example system.

FIG. 8 is a flowchart of an example method.

FIG. 9 is a flowchart of an example method.

FIG. 10 is a flowchart of an example method.

DETAILED DESCRIPTION

Examples of methods and systems are described herein. It should beunderstood that the words “exemplary,” “example,” and “illustrative,”are used herein to mean “serving as an example, instance, orillustration.” Any embodiment or feature described herein as“exemplary,” “example,” or “illustrative,” is not necessarily to beconstrued as preferred or advantageous over other embodiments orfeatures. Further, the exemplary embodiments described herein are notmeant to be limiting. It will be readily understood that certain aspectsof the disclosed systems and methods can be arranged and combined in awide variety of different configurations.

I. EXAMPLE SAMPLE IMAGING

It can be beneficial in a variety of applications to microscopicallyimage a sample. For example, the sample could include cultured humancells (e.g., cancer cells, normal cells) and imaging the sample couldfacilitate determining the effectiveness of a drug at eliminating cellsin the sample (e.g., the effectiveness of chemotherapy drugs ateliminating cancer cells), determining the toxicity of a substance(e.g., the toxicity of a chemotherapy drug to non-cancerous cells), ordetermining some other information about the contents of the sampleand/or about the effect of a substance on the contents of the sample.Imaging the sample can include using bright field microscopy,fluorescence microscopy, structured illumination, confocal microscopy,or some other imaging techniques to generate image data about thecontents of the sample.

Bright field microscopy has the benefit of being able to be performedwithout adding dyes, fluorophores or other labels (e.g., by addition ofthe labels and/or by transfection of sample contents to express thelabels) that can alter the ‘natural’ behavior of sample contents (suchimaging, without added labels, may be referred to as “label-free”imaging). Alternatively, fluorescent dyes or other labels can be addedto facilitate imaging of particular tissue structures and/orphysiological processes. For example, an Annexin V green fluorescentdye-based reagent could be added to image the location, rate, or otherinformation about cell death in a sample. In another example, a NucLightRed indicator could be added to image information about cellproliferation in a sample. In some examples, the cells in a sample couldbe genetically modified to express fluorescent proteins or other labelsrelated to processes of interest. For example, patient-derived neuralinduced pluripotent stem cells (iPSCs) could be genetically modified toexpress Aβ₁₋₄₂-GFP or some other fluorescently-tagged protein associatedwith Alzheimer's disease to facilitate the assessment of putativeAlzheimer's treatments.

Many objectives or other optical elements of an imaging apparatus usedto image samples may be limited with respect to depth of field. That is,the optical apparatus may be limited to imaging, in focus, light that isreceived from a relatively small-volume and/or substantially planarvolume of a sample. Light received from outside of this volume may bereceived (by a charge-coupled device or other light-sensitive element ofthe imaging apparatus) out of focus. This limitation may be related tothe cost or overall quality of the imaging apparatus, a desire to reducechromatic, axial, spherical, or other aberration in the imagingapparatus, a desire to reduce the volume, size, weight, or number ofparts of the imaging apparatus, a desire to improve the performance ofthe apparatus in some way by restricting its use to imaging small-volumeand/or substantially planar volumes, or due to some other factor(s).Such a limitation in the depth of field of an imaging apparatus may becompensated for in a variety of ways.

In some examples, the samples to be imaged may be intrinsicallysmall-volume and/or substantially planar or may be modified to be so.However, many samples of interest contain individual objects that mayspan a volume that does not fit within the narrow depth-of-field of animaging device and/or may contain useful information about thedistribution, interconnection, or other spatial information of multipleobjects across a distance within the sample, in a directionperpendicular to an imaging plane of the imaging device that exceeds thenarrow depth-of-field of the imaging device. Such ‘three-dimensional’samples may include organoids, tumor spheroids, or some otherthree-dimensional, multi-cellular structures of interest. Further, suchstructures may be cultured or otherwise disposed within a medium (e.g.,a dome or otherwise-shaped volume of extracellular medium) that spans avolume that does not fit within the narrow depth-of-field of an imagingdevice. For example, the sample could be part of a multi-spheroid assaywherein tumor spheroids (or some other variety of spheroids) arearranged in a layered and embedded format within a volume of matrigel.

Organoids (e.g., pancreatic-cell organoids, hepatic-cell organoids,intestinal-cell organoids) and tumor spheroids are of particularinterest, as their three-dimensional structure more closely mimics the‘natural’ three-dimensional environment of the cells being cultured.Accordingly, the reaction of organoids, tumor spheroids, or other suchthree-dimensional multi-cellular structures to drugs or other appliedexperimental conditions is likely to more closely mimic the response tocorresponding samples in the human body or other some other environmentof interest. Organoids may be cultured from a patient's own cells, inorder to predict the particular patient's response to a range ofdifferent possible treatments. For example, iPSCs could be extractedfrom the patient and used to culture a neuron organoid, a breast cancerorganoid, or organoids approximating some other healthy or non-healthy(e.g., cancerous) tissue of interest to facilitate assessment of theresponse of such tissues to possible treatments.

In order to use a limited depth-of-field imaging apparatus to image thecontents of such samples, a variety of techniques may be used. In someexamples, some or all of the sample could be spread onto a microscopeslide or other flat member to facilitate imaging of the sample. Howeversample-spreading or other sample preprocessing methods to make a samplecompatible with a limited depth-of-field imaging apparatus may not beapplicable to a sample of interest, may result in destruction of thesample, loss or distortion of information about the sample (e.g.,distortion of the sample due to interaction with a microtome, freezing,fixation, being removed from a sample container, being spread, etc.), orother unwanted effects, or may be inapplicable to the sample ofinterest. Further, such sample preprocessing methods preclude thepossibility of imaging the same sample (e.g., the same sample ofcultured organoids or tumor spheroids) at multiple points in time, e.g.,to analyze the long-term effects of a drug or other experimentalcondition over time.

Additionally or alternatively, an imaging apparatus with a more shallowdepth of field could be replaced with an imaging apparatus having adeeper depth of field; however, such an improved imaging apparatus couldbe too costly, could be too large to fit inside an incubator or otherenvironment of interest, or could be undesirable with respect to someother consideration.

The above limitations of a limited-depth-of-field imaging apparatus maybe fully or partially alleviated by employing the methods describedherein. Applying these methods to the operation of such a limiteddepth-of-field imaging apparatus allows for the generation of imageinformation that could alternatively be obtained from an imagingapparatus having a depth-of-field equal to or exceeding the volumespanned by the sample in a direction parallel to the optical axis of theimaging apparatus. The image processing methods described hereinincludes generating a plurality of images of a ‘three-dimensional’sample (which may be referred to as a ‘stack’ of images, each imagecorresponding to a respective different depth within the sample). Eachimage could be taken with a respective different depth within the samplebeing in focus by using a motor or other actuator to control thelocation of the imaging apparatus relative to the sample. This couldinclude moving the camera, moving the sample container, or moving bothof the sample container and the camera.

FIG. 1 illustrates an example system 100 that could be used to generatesuch a set of images. The system 100 includes an imaging apparatus 110(e.g., a camera, objective, and/or other optical or electrical elements)and a light source 120 that can be used to image a sample contained in asample container 130. The sample container 130 could be a well of amulti-well incubation or culture plate (e.g., a 96-well culture plate).The sample container 130 contains a number of objects 105 (e.g.,organoids, tumor spheroids, or other three-dimensional biologicalobjects) contained within a sample medium 135 (e.g., a dome ofextracellular medium used to culture the organoids prior to transferringthe organoids to wells of a 96-well multi-well culture plate or someother multi-well culture plate prior to application of a variety ofdiffering putative treatments or other experimental conditions, or amulti-spheroid assay that includes a layered and embedded arrangement oftumor spheroids within a volume of matrigel).

The arrangement of the elements (e.g., camera 110 and light source 120)in FIG. 1 is intended as a non-limiting example. A light source used toilluminate a sample could be located opposite the sample from a cameraor other imaging apparatus, on the same side of the sample as theimaging apparatus, to the side of the sample relative to the imagingapparatus, or at some other location relative to the sample and imagingapparatus. Further, a light source could be incorporated into an imagingapparatus and/or could share one or more optical elements or opticalpaths with an imaging apparatus (e.g., an objective of a confocalimaging system could be shared between a laser or other element used toilluminate the sample and a charge-coupled device or other element usedto image light received from the sample). In some examples, multiplelight sources could be provided in multiple locations, e.g., tofacilitate operating to image the sample according to different modes(e.g., fluorescent imaging, brightfield imaging, reflective/scatteringvisible light imaging, structured illumination imaging, phase contrastimaging).

The camera 110 and/or light source 120 could be configured such that thecamera 110 is capable of imaging light received from a small,substantially flat volume within the sample container 130. Such aconfiguration could be chosen instead of a large depth-of-fieldconfiguration in order to reduce cost, increase reliability, reduce sizeor mass, or to provide some other benefit(s). For example, such aconfiguration could be chosen to reduce to size and/or weight of thecamera in order to facilitate mounting the camera 110, light source 120,and/or other elements of an imaging apparatus on an actuated gantrywithin an incubator so as to allow for long-term automated imaging ofmultiple samples within an incubator at multiple different points intime. Region 140 a represents, in cross-section, the extent of a firstexample volume that could be imaged in-focus by such a limiteddepth-of-field imaging apparatus. Region 140 b represents, incross-section, another example of such a region, differing from thefirst region 140 a with respect to depth within the sample container 130but spanning substantially the same area in directions perpendicular toan optical axis of the camera 110.

The location of the camera 110 relative to the sample container 130could be set, at different points in time, to facilitate imaging arespective different volumes, or depths, within the sample container130. This could include using an actuated gantry or other means to movethe camera 110 relative to the sample container 130 (e.g., by moving thecamera 110 and/or moving the sample container 130) in order to select aflat region within the sample container 130 to image in focus. Forexample, during a first period of time (e.g., a period of timeillustrated in FIG. 1), the camera 110 could operate to image the firstvolume 140 a in focus. The camera 110 could then be moved downwardrelative to the sample container 130 and subsequently operated to imagethe second volume 140 b in focus. Such a procedure could be repeated fora plurality of different volumes, and corresponding depths, within thesample container 130 in order to generate a set or ‘stack’ of imagesthat represents the contents of the sample container 130. Such aprocedure can facilitate the generation of in-focus image informationfor objects 105 (e.g., organoids, tumor spheroids, or otherthree-dimensional biological objects) that have a characteristicdimension greater than the depth of field of the camera 110 and/or thatare located at points ranging along the vertical dimension within thesample container 130 at distances greater than the depth of field of thecamera 110.

Such a process to generate sets of images that are ‘stacked’ could beperformed in an automated manner. For example, a camera and associatedactuator could operate to generate such a set of images of a sample at anumber of specified number of points in time (e.g., once every hour,once every 24 hours, etc. to facilitate analysis of the response of thesample contents to an applied experimental condition over time).Additionally or alternatively, the sample could be located within amulti-well sample container, and a set of images could be generated foreach of the samples in the container by actuating the camera in twodimensions to select a particular sample (e.g., by operating actuatorsof a gantry that contains the camera) and in a third dimension to imagedifferent depths within the particular sample.

FIG. 2A shows an example set (or ‘stack’) of images 200 of a sample.Each image of the set of images 200 corresponds to a respective focalplane within the sample. Accordingly, each image may represent, infocus, respective different contents of the sample depending on thelocation of the contents relative to the focal plane corresponding tothe image. In a first example, a top image 210 a of the set of images200 corresponds to a first focal plane at which none of the samplecontents are located. Accordingly, the image information of the topimage 210 a is all out-of-focus. In another example, a second image 210b of the set of images 200 corresponds to a second, different focalplane at which some of the sample contents are located. Accordingly,some of the image information of the second image 210 a (i.e., thethree-dimensional object to the left and bottom of the second image) isin focus, while other portions of the image information of the secondimage 210 b are out-of-focus.

Such a set of images 200 can contain sufficient in-focus imageinformation to generate a projection image of the sample that providesan improved view of the distribution of the contents within the sample.For example, some or all of the contents of the sample may berepresented in-focus in the projection image. The projection image couldapproximate an image of the sample as if it had been generated using awider depth of field FIG. 2B shows an example of such a projection image250 that has been generated from a set of images including the set ofimages 200 shows in FIG. 2A. Such a projection image could facilitateimproved imaging and analysis of the contents of a sample. For example,a projection image could be segmented in order to identify discretecells, organoids, tumor spheroids, or other three-dimensional contentsof the projection image. Such an image segmentation could then be usedto automatically determine a number, size, identity, morphology, orother information about cells, organoids, tumor spheroids, or some otheranalysis of the contents of the sample. Additionally or alternatively,such a projection image could facilitate subjective analysis of thecontents of the sample by a pathologist, a researcher, anepidemiologist, or some other person.

A projection image could be generated for a sample, from a set of imagescorresponding to respective different focal planes within the sample, ina variety of ways. In some examples, this could include using the set ofimages to determine the depth at which contents of the sample arelocated and/or come into focus. Pixels or other image information fromthe image, or the set of images, that corresponds to the determineddepth could then be used (or ‘projected’) to generate correspondingpixels of the projection image. Such depth information couldadditionally or alternatively be used to generate a depth map forcontents of the sample (e.g., to facilitate analysis of thethree-dimensional geometry and/or arrangement of the contents of thesample) and/or to improve segmentation of the projection image (or ofsome other image of the sample).

II. EXAMPLE GENERATION OF DEPTH MAPS

Sets of narrow depth-of-field images (e.g., bright field images) of asample, corresponding to respective different depths spanning a range ofdepths within the sample, may contain sufficient information todetermine information about the depth of contents of the sample. Suchsets of ‘stacked’ images may also contain sufficient image informationthat can be combined with the depth information to generate a projectionimage of the sample that represents the contents of the sample in-focusacross a range of depths (e.g., as a sort of simulated widedepth-of-field image of the sample). The methods described hereinfacilitate generating such depth information (e.g., as depth maps) andsuch projection images from sets of narrow field-of-view images (e.g.,brightfield images, fluorescence images).

These methods include detecting edges, textures, or otherhigh-spatial-frequency contents within each image of the set of images.The presence of such high-spatial-frequency contents at a particularlocation within a particular image may indicate that that the contentsof the sample at the particular location were imaged in-focus.Accordingly, it can be assumed that the contents of the sample at theparticular location are located at the depth, within the sample, thatcorresponds to the depth of the particular image. Thus, a depth map orother depth information may be generated for a sample by identifying‘in-focus’ regions within each image of a set of images of the sample.These identified regions can then be combined across the set of imagesto generate a single depth map and/or projection image of the sample.

Identifying ‘in-focus’ regions within each image of a set of images ofthe sample can include applying a filter or transformation to each ofthe images. For example, a Canny edge detector or other edge detectionfilter or algorithm could be applied to generate, for each image in theset of images, a respective ‘edge image’ that represents where edges arelocated in each of the images. In another example, a texture filtercould be applied to each image in the set of images to generaterespective ‘texture images’ that represents where, in each image,regions of increased high-spatial-frequency information (or ‘texture’)are located.

“Texture” can be determined in a variety of ways. For example, a texturevalue for a particular pixel of an image can be determined bydetermining an entropy, a numerical range, a standard deviation, avariance, a coefficient or variation, or some other measure of thevariability of a set of pixels in the neighborhood of the particularpixel. Such a neighborhood could be a square or otherwise shaped regionof neighboring pixels, e.g., a five-by-five square of pixels centered onthe particular pixel for which the texture value is being determined.The pixels in the ‘neighbor’ region could be equally weighted, or couldbe used in a weighted manner to determine a texture value (e.g., bygiving higher weighting to pixels closer to the particular pixel forwhich the texture value is being determined than to farther away pixelswhen determining, e.g., an entropy, a standard deviation, etc.). Suchtexture values could be determined for every pixel in an image, for asubset of the pixels (e.g., for every other pixel) in the image, or forsome other set of locations within the image.

As noted above, regions of a particular narrow depth-of-field image thathave higher texture values (or higher values of some other property thatis related to high-spatial-frequency contents of an image) are morelikely to have been imaged in-focus. Thus, contents of a sample thatcorrespond to a particular high-texture location of a particular imageof the sample are likely to be located at a depth within the samplecorresponding to the depth of the focal plane of the particular image.The texture information for a particular location within the set ofimages (e.g., corresponding to a location of a particular pixel indexwithin each of the images) can be compared across all of the images inorder to determine a single depth value for the particular location.Such a determination could be used to generate a full depth map for thesample and/or to generate pixels of a projected output image of thesample (e.g., by selecting pixel(s) of the image, of the set of images,that corresponds to the determined depth for use in generating pixels ofthe projection image).

To illustrate this process, FIG. 3A shows an image 301 (e.g., aprojection image generated as described herein) of a sample. The image301 is shown next to a depth map 300 of the sample. Each object in theimage 301 may include one or more organoids, tumor spheroids, or otherthree-dimensional multi-cellular objects. The depth map 300 shows thatsome of the apparently singular objects depicted in the image 301possibly depict multiple overlapping objects at differing depths withinthe sample.

FIG. 3B illustrates the depth map 300 of the sample and a set of narrowdepth-of-field images 310 a, 310 b, 310 c, 310 d of the sample thatcorrespond to respective different focal planes at respective differentdepths within the sample. The depth map 300 and images of the set ofimages have the same resolution and pixels locations for ease ofexplanation. It will be appreciated by one of skill in the art that themethods herein may be adapted to circumstances where the depth map,projection image, and/or input images of the set of images havediffering resolutions and/or pixel locations.

A depth value for a particular pixel 305 of the depth map 300 may bedetermined based on the texture values (or other values representing thelocal magnitude of high-spatial-frequency image content) ofcorresponding pixels 315 a, 315 b, 315 c, 315 d in each image of the setof images 310 a, 310 b, 310 c, 310 d. As shown in FIG. 3B, the images ofthe set of images and the depth map may have the same size andresolution, in which case the pixel correspondences may be one-to-one.This is intended as a non-limiting example of determining a depth valuefor a particular pixel of a depth map based on texture information fromeach of the input images at locations, within the input images, thatcorrespond to the location of the particular pixel within the depth map.

The depth value may be selected, based on the set of texture (or otherhigh-spatial-frequency image content) values, in a variety of ways. Forexample, the highest texture value (or lowest, if lower texture valuescorrespond to greater amounts of high-spatial-frequency image content)could be determined and the depth determined according to the depth ofthe input image corresponding to the highest texture value. This couldbe done based on the assumption that the highest texture value is likelyto correspond, for a particular pixel location, to the image that ismost in-focus at that particular location and thus that the contents ofthe sample are likely to be located at the corresponding depth withinthe sample.

Additional or alternative methods could be applied to determine a depthvalue based on the set of texture values. For example, the depth may bedetermined based on the depth of a peak or other feature detected withinthe set of texture values. FIG. 3C illustrates a plot of a set oftexture values 320 determined for a particular pixel of an output depthmap as a function of the depth of the input images used to generate thetexture values. A peak 325 is present in the texture data; this peakcould be detected and the depth (‘d₁’ in FIG. 3C) determined therefrom.Peak detection could be employed to determine the depth value based onthe assumption that peaks within the texture values as a function ofdepth are likely to correspond, for a particular pixel location, to thecontents of the sample being in-focus at that particular location andthus that the contents of the sample are likely to be located at thecorresponding depth within the sample.

Depth values determined as described above may be used as-is as pixelsof a depth map, to generate pixels of a projection image, to improve thesegmentation of one or more images of a sample, or to facilitate someother application. Alternatively, some level of spatial pre-processingmay be applied to the determined depth values prior to such applications(e.g., spatial pre-processing could be applied to a set of depth valuesdetermined as described above in order to generate a depth map). Forexample, a two-dimensional low-pass or other type of linear filter couldbe applied to the depth values prior to applying them in an application.Additionally or alternatively, a non-linear pre-processing method couldbe applied. For example, an edge-preserving low-pass spatial filtercould be applied. As another example, depth values that are outliersrelative to their neighbors (e.g., that exceed the mean value for theirneighbors by more than a specified amount, e.g., a multiple of thestandard deviation of their neighbors) could be removed, set to aparticular amount (e.g., a mean, median, or other central measure oftheir neighbors), filtered using more aggressive filter parameters, orpre-processed in some other manner to reduce the effect of such outlierson subsequent processing.

III. EXAMPLE IMAGE FORMATION

Depth information for a sample (e.g., a depth map, individual depthvalues) may be used to project pixels or other image information from aset of images of the sample into a single projection image of the sample(e.g., as in example projection images 250 and 301). The depthinformation can be used to determine which image(s) of the set of imagesto draw from when setting the intensity, color, or other imageinformation for each pixel of the projection image. As noted above,where the set of input images varies with respect to focal plane withinthe sample, depth information can be used to select which of the imagesto project from when generating pixels of the projection image such thatthe projection image appears completely in focus or otherwise improvedrelative to the set of input images.

FIG. 4 illustrates a projection image 400 of a sample and a set ofnarrow depth-of-field images 410 a, 410 b, 410 c, 410 d of the samplethat correspond to respective different focal planes at respectivedifferent depths within the sample. The projection image 400 and imagesof the set of images have the same resolution and pixels locations forease of explanation. It will be appreciated by one of skill in the artthat the methods herein may be adapted to circumstances where the depthmap, projection image, and/or input images of the set of images havediffering resolutions and/or pixel locations.

An image value (e.g., one or more of a luminance, a chrominance, a red,a green, and/or a blue channel color value) for a particular pixel 405of the projection image 400 may be determined based on a depth valuedetermined for the particular pixel 405. Such a depth value may beobtained from a depth map of the sample, or may be determined on apixel-by-pixel basis. The depth value may be determined as describedabove (e.g., by identifying the depth of an image, of a set of images,that has the highest texture value at a location corresponding to eachpixel of the projection image), or using some other method, e.g., usinga phase contrast image, a depth sensor, of some other depth detectingmeans.

Determining the image value for the particular pixel 405 may includecopying the image value of a corresponding pixel of an image thatmatches the depth value for the particular pixel 405. For example, inFIG. 4 the second image 410 b has a depth corresponding to the depthvalue determined for the particular pixel 405. Accordingly, the imagevalue (e.g., luminance, intensity, etc.) of the corresponding pixel 415b of the second image 410 b is copied, or projected, to be the imagevalue for the particular pixel 405.

This one-to-one projection of image values is intended as a non-limitingexample of determining an image value for a particular pixel of aprojection image based on a depth value and one or more narrowdepth-of-field images corresponding to that depth value. Additionalpixels could be used (e.g., combined in a weighted combination) togenerate the image value for the particular pixel 405. In some examples,the image value of the particular pixel 405 and/or of neighboring pixelscould be determined, in whole or in part, based on a number of pixelsneighboring the corresponding pixel 415 b of the second image 410 b.Additionally or alternatively, image information from multiple imagesthat correspond to depth values within a specified range of the depthvalue of the particular pixel 405 may be used to generate the imagevalue of the particular pixel 405. For example, the image value of theparticular pixel 405 could be determined based on a combination of thecorresponding pixel 415 b of the second image 410 b and correspondingpixels 415 a, 415 c of the first 410 a and third 410 c images, the first410 a and third 410 c images corresponding to depths within aneighborhood (e.g., a specified range) of the depth value of theparticular pixel 405.

IV. EXAMPLE IMAGE SEGMENTATION

It can be beneficial in a variety of contexts to identify, in anautomated manner, the extent, location, size, identity, and/or otherinformation about organoids, tumor spheroids, cells, particles, or otherthree-dimensional, multi-cellular discrete contents within an image of abiological sample or other environment of interest. Such a process maybe referred to as “image segmentation.” Image segmentation can be usedto automatically perform a variety of analyses on the content of animage, e.g., to determine a number, type, volume/size, spatialdistribution, shape, growth rate, or other properties of cells,organoids, or tumor spheroids in a sample.

A variety of methods are available in the art to perform segmentation onmicroscopic images of biological samples. Such methods can include oneor more of thresholding, clustering, edge detection, region-growing, anartificial neural network or other machine-learning algorithm, or someother technique or combination of techniques to identify putativelydifferent contiguous regions within an image. Each region identifiedusing such methods could correspond to a respective organoid, cancerspheroid, cell, or component or portion thereof. Such methods could beapplied to one or more narrow depth-of-field images of a sample. Thesegmentation could be improved by segmenting a projection image of asample, determined as described above, since such a projection imagerepresents more of the contents of the sample in-focus than anyindividual narrow depth-of-field image used to generate the projectionimage.

FIG. 5A shows an example image 500 of a sample. The image could be aprojection image generated as described elsewhere herein or could beobtained in some other manner. As illustrated in FIG. 5B, the image 500can be segmented to generate a first segmentation map 510 of the sample.The regions include first 515 a, second 515 b, and third 515 c exampleregions. Each region could represent a respective different object,multiple objects, and/or portions of objects (e.g., organoids, tumorspheroids, cells) within the sample.

The accuracy of the identified regions can be limited by the imageinformation that is available in the input image 500. For example, theboundaries, within the input image, between separate overlapping oradjacent objects may be fuzzy or otherwise sub-optimal in a manner thatprevents the segmentation method from identifying the separate objectsas respective different regions in the segmentation map. FIG. 5C shows adepth map 520 of the sample that is represented by the input image 500.The depth map 520 may be generated based on a set of narrowdepth-of-field images using the methods described herein (e.g., based ontexture information present in the set of images) or may be generatedvia some other method (e.g., using a depth sensor). The depth map 520shows that at least the first 515 a and second 515 b regions of thefirst segmentation map 510 likely represent multiple distinguishableobjects, based on the difference in depth between different contiguousregions within the areas 525 a, 525 b of the depth map 520 thatcorrespond to the first 515 a and second 515 b regions, respectively.

Accordingly, depth information provided by a depth map 520 or from someother source may be applied to improve the segmentation of an image.This can include using the depth information from the depth map tofurther divide one or more regions of the first segmentation image togenerate an improved second segmentation image. For example, the depthinformation corresponding to a particular region of the firstsegmentation map 510 could be analyzed to determine whether itrepresents more than one potentially discrete population of depthvalues. If it does, the particular region could be further dividedaccording to the locations of the depth values within each of thediscrete populations.

As an example, FIGS. 5C and 5D illustrate an analysis of the first 515 aand second 515 b regions of the first segmentation map 510 to determinewhether to further divide those regions when generating an improvedsecond segmentation map 530. For the first region 515 a, depth values(e.g., pixels of the depth map 520) that correspond to the location andextent of the first region 515 a (indicated in FIG. 5C as a first patch525 a of depth values) can be analyzed to identify regions into whichthe first region 515 a can be further divided.

This could include performing clustering, region growing, or otheranalyses on the population of depth values within the first patch 525 ato identify two or more regions within the first patch 525 a. The firstregion 515 a can then be further divided, into regions 535 a, 535 b, and535 c, of the second segmentation map 530. This division can includegenerating the additional divisions based on the locations of the depthmap pixels of the depth map 520 that correspond to each of theidentified clusters. Such a process could include performing filtering,region-growing, edge-preservation, or other processes to ensure that thedetermined divisions result in contiguous and/or reasonably smoothregions 535 a, 535 b, and 535 c following the division of the parentregion 515 a. A similar process could be applied to divide the secondregion 515 b of the first segmentation map 510 into correspondingregions 535 d, 535 e of the second segmentation map 530. Note that suchanalysis can also result in no division being performed, as the thirdregion 515 c of the first segmentation map 510 has been maintained as asingle region 535 f in the second segmentation map 530.

V. EXAMPLE APPLICATIONS

The systems and methods described herein can be used to facilitate avariety of biological applications in sample imaging. This can includeimaging, at a plurality of points in time across hours, days, weeks, orsome other duration, or a plurality of samples (e.g., 96 samplescontained within respective wells of a 96-well sample plate) that arelocated within an incubator using an automated imaging system to avoidhaving to perturb the samples by removing them from the incubator. Theimaged samples could contain 3D cultures of human cells or tumor cells,organoids, tumor spheroids, or other cells. The cells could be natural,or could be the result of some experimental process (e.g., the result ofexposure to a carcinogen in order to create cancer cells). Further, thecells could be labeled with a fluorescent dye or genetically modified toexpress a fluorescent protein, other reporter substance, or to providesome other biological insight (e.g., to assess the effects of thegenetic modification on the cells).

The samples could be prepared and imaged in order to perform drugdiscovery and/or assess the toxicology of a drug or other substance inthe fields of immunology, oncology, neuroscience, cell therapy, or otherfields of research. In examples where the samples include and/or arepermitted to develop organoids, imaging the samples can facilitateinvestigation of organ development, the development and/orquantification of disease models, and/or the development of regenerativemedicine.

The described systems and methods support a variety of imagingmodalities. For example, label-free, bright-field images could be takenof organoids, multi-spheroids embedded in extracellular medium, or otherthree-dimensional samples that may be, for example, disposed in wells ofa 96-well assay. Fluorescence imaging could be employed to imagefluorescent reporters that may act to label cells and/or may bereporters of cell function or some other property of interest as part ofa fluorescent assay. For example, an Annexin V green fluorescentdye-based reagent could be imaged to assess the location, rate, or otherinformation about cell death in a sample, a NucLight Red indicator couldbe imaged to assess information about cell proliferation in a sample, orsome other fluorescent reporter or system of reporters (e.g., amulti-color FUCCI assay) could be fluorescently imaged to assessinformation about cell division, function, identity, or differentiation.This could be done in order to, e.g., perform a cell health assay ofsamples containing organoids, multi-spheroids, or some otherthree-dimensional objects of interest.

Where multiple different samples in a multi-well plate (e.g., a 96-wellplate) are imaged, the samples could vary from well to well with respectto cell contents (e.g., cell types, tumor cell types), the amount oridentity of an added substance (e.g., a dose of an added drug, aparticular variety of an added drug variant as part of a drug discoveryassay), a type of genetic modification, or some other varyingexperimental condition. The automated imaging and image processingtechniques described herein could then be applied to assess the efficacyand/or toxicity of applied substances/treatments, or to determine someother experimental data of interest.

VI. EXAMPLE SYSTEMS

A variety of systems may be employed (e.g., programmed) to perform thevarious embodiments described herein. Such systems can include desktopcomputers, laptop computers, tablets, or other single-user workstations.Additionally or alternatively, the embodiments described herein may beperformed by a server, cloud computing environment, or other multi-usersystem.

Such systems could analyze data received from other systems, e.g., datareceived from a remote data storage on a server, from a remote cellcounter or other instrument, or from some other source. Additionally oralternatively, a system configured to perform the embodiments describedherein may include and/or be coupled to an automated incubator, sampleimaging system, or some other instrument capable of generatingexperimental data for analysis. For example, such an instrument couldinclude an incubator that contains a multi-well sample container. Thesamples within such a multi-well sample container could differ withrespect to the genome of the samples, the source of the samples, thegrowth medium applied to the samples, a pharmaceutical or biologicapplied to the samples, or some other condition applied to the samples.

Samples within such an apparatus could be experimentally assessed in avariety of ways. The samples could be imaged (e.g., using visible,infrared, and/or ultraviolet light). Such imaging could includefluorescent imaging of the contents of the samples, e.g., imagingfluorescent dyes or reporters added to the samples and/or generated bythe cells of the samples (e.g., following insertion of genes coding forfluorophores). An automated gantry could be located within such anincubator to facilitate imaging of the various samples within respectivewells of the sample container or to facilitate the measurement andanalysis of the various samples within respective wells of the samplecontainer.

As an example, an automated imaging system may be employed to obtain, inan automated fashion, images (e.g., brightfield images, fluorescenceimages) of a plurality of biological samples, in respective wells of asample container, during a plurality of different scan periods overtime. A set of images could be taken, by the automated imaging system,of each sample during each of the scan periods, e.g., a set of imagesdiffering with respect to focal plane within the sample. The images canthen be analyzed in order to determine a depth map, a projection image,or some information about the samples, e.g., according to the methodsdescribed herein.

Use of such an automated imaging system can significantly reduce thepersonnel costs of imaging biological samples, as well as increasing theconsistency, with respect to timing, positioning, and image parameters,of the images generated when compared to manual imaging. Further, suchan automated imaging system can be configured to operate within anincubator, removing the need to remove the samples from an incubator forimaging. Accordingly, the growth environment for the samples can bemaintained more consistently. Additionally, where the automated imagingsystem acts to move a microscope or other imaging apparatus relative tothe sample containers (instead of, e.g., moving the sample container tobe imaged by a static imaging apparatus), movement-related perturbationof the samples can be reduced. This can improve the growth anddevelopment of the samples and reduce movement-related confounds.

Such an automated imaging system can operate to obtain one or moreimages during scans that are separated by more than twenty-four hours,by more than three days, by more than thirty days, or by some longerperiod of time. The scans could be specified to occur at a specifiedrate, e.g., once per daily, more than daily, more than twice daily, ormore than three times daily. The scans could be specified such that atleast two, at least three, or some greater number of scans occurs withina twenty-four hour period. In some examples, data from one or more scanscould be analyzed (e.g., according to the methods described herein) andused to determine the timing of additional scans (e.g., to increase arate, duration, image capture rate, or some other property of the scansin order to detect the occurrence of a discrete event that is predictedto occur within a sample).

The use of such an automated imaging system can facilitate imaging ofthe same biological sample at multiple points in time over long timeperiods. Accordingly, the development and/or behavior of individualcells and/or networks of cells (e.g., organoids, tumor spheroids) can beanalyzed over time. For example, a set of cells, portions of cells, orother objects could be identified, within a single sample, within scanstaken during different, widely spaced periods of time. These sets ofidentified objects could then be compared between scans in order toidentify the same object(s) across the scans. Thus, the behavior ofindividual organoids, tumor spheroids, cells, or portions of cells, canbe tracked and analyzed across hours, days, weeks, or months.

FIG. 6 illustrates elements of such an automated imaging system 600. Theautomated imaging system 600 includes a frame 610 to which otherelements of the automated imaging system 600 are attached. The frame 610may be configured (e.g., sized) in order to fit within an incubator. Theautomated imaging system 600 includes a sample container 620 that isremovably placed within a sample container tray 630 that is coupled tothe frame 610. The sample container tray 630 could be removable and/orcould include a removable insert to facilitate holding a variety ofdifferent sample containers (e.g., a variety of industry-standard samplecontainers). The system 600 additionally includes an actuated gantry 650configured to position an imaging apparatus 640 relative to the samplecontainer 620 such that the imaging apparatus 640 can operate to obtainimages of the contents of individual wells of the sample container 620(e.g., the example well 625).

The imaging apparatus 640 can include a microscope, a fluorescenceimager, a two-photon imaging system, a phase-contrast imaging system,one or more illumination sources, one or more optical filters, and/orother elements configured to facilitate imaging samples contained withinthe sample container 620. In some examples, the imaging apparatus 640includes elements disposed on both sides of the sample container 620(e.g., a source of coherent, polarized, monochromatic, orotherwise-specified illumination light in order to facilitate, e.g.,phase contrast imaging of biological samples). In such examples,elements on both sides of the sample container 620 may be coupled torespective different gantries, to the same gantry, and/or elements onone side of the sample container 620 may not be movable relative to thesample container 620.

The actuated gantry 650 is coupled to the frame 610 and the imagingapparatus 640 and configured to control the location of the apparatus640 in at least two directions, relative to the sample container 620, inorder to facilitate imaging of a plurality of different samples withinthe sample container 620. The actuated gantry 650 may also be configuredto control the location of the imaging apparatus 640 in a thirddirection, toward and away from the sample container 620, in order tofacilitate controlling the focal distance of images obtained using theimaging apparatus 640 and/or to control a depth of material, within thesample container 620, that can be imaged using the imaging apparatus640. Additionally or alternatively, the imaging apparatus 640 mayinclude one or more actuators to control a focal distance of the imagingapparatus 640. The imaging apparatus 640 could include one or moremotors, piezo elements, liquid lenses, or other actuators to facilitatecontrolling the focus setting of the imaging apparatus 640. For example,the imaging apparatus 640 could include an actuator configured tocontrol a distance between the imaging apparatus 640 and a sample beingimaged. This could be done in order to ensure that the image is takenin-focus and/or to allow images to be taken such that a variety ofdifferent focal planes within the sample are represented in respectivedifferent images.

The actuated gantry 650 may include elements configured to facilitatedetection of the absolute and/or relative location of the imagingapparatus 640 relative to the sample container 620 (e.g., to particularwell(s) of the sample container 620). For example, the actuated gantry650 may include encoders, limit switches, and/or other location-sensingelements. Additionally or alternatively, the imaging apparatus 640 orother elements of the system may be configured to detect fiducial marksor other features of the sample container 620 and/or of the samplecontainer tray 630 in order to determine the absolute and/or relativelocation of the imaging apparatus 640 relative to the sample container620.

Computational functions (e.g., functions to operate the actuated gantry650 and/or imaging apparatus 640 to image samples within the samplecontainer 620 during specified periods of time and/or to perform someother method described herein) may be performed by one or more computingsystems. Such a computing system may be integrated into a laboratoryinstrument system (e.g., 600), may be associated with such a system(e.g., by being connected via a direct wired or wireless connection, viaa local network, and/or via a secured connection over the internet),and/or may take some other form (e.g., a cloud computing system that isin communication with an automated imaging system and/or that has accessto a store of images of biological samples).

FIG. 7 illustrates an example of such a computing system 700, which maybe used to implement the methods described herein. The example computingsystem 700 includes a communication interface 702, a user interface 704,a processor 706, one or more sensors 707 (e.g., photodetectors, cameras,depth sensors, a microscope, or some other instrumented laboratoryapparatus), and data storage 708, all of which are communicativelylinked together by a system bus 710.

The communication interface 702 may function to allow the computingsystem 700 to communicate, using analog or digital modulation ofelectric, magnetic, electromagnetic, optical, or other signals, withother devices, access networks, and/or transport networks. Thus,communication interface may facilitate circuit-switched and/orpacket-switched communication, such as plain old telephone service(POTS) communication and/or Internet protocol (IP) or other packetizedcommunication. For instance, communication interface 702 may include achipset and antenna arranged for wireless communication with a radioaccess network or an access point. Also, communication interface 702 maytake the form of or include a wireline interface, such as an Ethernet,Universal Serial Bus (USB), or High-Definition Multimedia Interface(HDMI) port. Communication interface may also take the form of orinclude a wireless interface, such as a WiFi, BLUETOOTH®, globalpositioning system (GPS), or wide-area wireless interface (e.g., WiMAXor 3GPP Long-Term Evolution (LTE)). However, other forms of physicallayer interfaces and other types of standard or proprietarycommunication protocols may be used over communication interface 702.Furthermore, communication interface 702 may comprise multiple physicalcommunication interfaces (e.g., a WiFi interface, a BLUETOOTH®interface, and a wide-area wireless interface).

In some embodiments, the communication interface 702 may function toallow computing system 700 to communicate with other devices, remoteservers, access networks, and/or transport networks. For example, thecommunication interface 702 may function to transmit and/or receive anindication of images of biological samples (e.g., sets of brightfield orother types of images that differ with respect to focal plane imagedwithin a sample) or some other information.

The user interface 704 of such a computing system 700 may function toallow computing system 700 to interact with a user, for example toreceive input from and/or to provide output to the user. Thus, userinterface 704 may include input components such as a keypad, keyboard,touch-sensitive or presence-sensitive panel, computer mouse, trackball,joystick, microphone, and so on. User interface 704 may also include oneor more output components such as a display screen which, for example,may be combined with a presence-sensitive panel. The display screen maybe based on CRT, LCD, and/or LED technologies, or other technologies nowknown or later developed. User interface 704 may also be configured togenerate audible output(s), via a speaker, speaker jack, audio outputport, audio output device, earphones, and/or other similar devices.

In some embodiments, user interface 704 may include a display thatserves to present video or other images to a user (e.g., video of imagesgenerated during a particular scan of a particular biological sample).Additionally, user interface 704 may include one or more buttons,switches, knobs, and/or dials that facilitate the configuration andoperation of the computing device. It may be possible that some or allof these buttons, switches, knobs, and/or dials are implemented asfunctions on a touch- or presence-sensitive panel. The user interface704 may permit a user to specify the types of samples contained withinan automated imaging system, to specify a schedule for imaging or otherassessment of the samples, to specifying parameters of imagesegmentation, event analysis, and/or some other analysis to be performedby the system 700, or to input some other commands or parameters foroperation of an automated laboratory system and/or for analysis of datagenerated thereby.

Processor 706 may comprise one or more general purpose processors—e.g.,microprocessors—and/or one or more special purpose processors—e.g.,digital signal processors (DSPs), graphics processing units (GPUs),floating point units (FPUs), network processors, tensor processing units(TPUs), or application-specific integrated circuits (ASICs). In someinstances, special purpose processors may be capable of imageprocessing, image alignment, statistical analysis, filtering, or noisereduction, among other applications or functions. Data storage 708 mayinclude one or more volatile and/or non-volatile storage components,such as magnetic, optical, flash, or organic storage, and may beintegrated in whole or in part with processor 706. Data storage 708 mayinclude removable and/or non-removable components.

Processor 706 may be capable of executing program instructions 718(e.g., compiled or non-compiled program logic and/or machine code)stored in data storage 708 to carry out the various functions describedherein. Therefore, data storage 708 may include a non-transitorycomputer-readable medium, having stored thereon program instructionsthat, upon execution by computing device 700, cause computing device 700to carry out any of the methods, processes, or functions disclosed inthis specification and/or the accompanying drawings. The execution ofprogram instructions 718 by processor 706 may result in processor 706using data 712.

By way of example, program instructions 718 may include an operatingsystem 722 (e.g., an operating system kernel, device driver(s), and/orother modules) and one or more application programs 720 (e.g., filteringfunctions, data processing functions, statistical analysis functions,image processing functions, depth determination functions, imagesegmentation functions) installed on computing device 700. Data 712 mayinclude microscopy images or other data that includes sets of images ofindividual samples, depth information for samples, and/or segmentationinformation for samples.

Application programs 720 may communicate with operating system 722through one or more application programming interfaces (APIs). TheseAPIs may facilitate, for instance, application programs 720 receivinginformation via communication interface 702, receiving and/or displayinginformation on user interface 704, and so on.

Application programs 720 may take the form of “apps” that could bedownloadable to computing device 700 through one or more onlineapplication stores or application markets (via, e.g., the communicationinterface 702). However, application programs can also be installed oncomputing device 700 in other ways, such as via a web browser or througha physical interface (e.g., a USB port) of the computing device 700.

In some examples, portions of the methods described herein could beperformed by different devices, according to an application. Forexample, different devices of a system could have different amounts ofcomputational resources (e.g., memory, processor cycles) and differentinformation bandwidths for communication between the devices. Forexample, a first device could be an embedded processor(s) that couldoperate an actuated gantry, imaging apparatus, or other elements togenerate information about biological samples at and/or during aplurality of different periods. A second device could then receive(e.g., via the internet, via a dedicated wired link), from the firstdevice, information (e.g., image information, depth information) fromthe first device and perform the processing and analysis methodsdescribed herein on the received data. Different portions of the methodsdescribed herein could be apportioned according to such considerations.

VII. EXAMPLE METHODS

FIG. 8 is a flowchart of a method 800 for generating a projection imageof a three-dimensional sample. The method 800 includes obtaining a setof images of the sample, wherein each image of the set of imagescorresponds to a respective focal plane within the sample (810). Themethod 800 additionally includes applying a filter to each image of theset of images to determine a respective depth value for each pixel of anoutput image of the sample, wherein a given depth value represents adepth, within the sample, at which the contents of the sample can beimaged in-focus (820). The method 800 additionally includes determiningan image value for each pixel of the output image based on the depthvalue of the pixel of the output image, wherein determining an imagevalue for a particular pixel of the output image comprises: (i)identifying an image of the set of images that corresponds to the depthvalue of the particular pixel; and (ii) determining the image value forthe particular pixel based on a pixel, of the identified image, having alocation within the identified image that corresponds to the particularpixel (830). The method 800 could include additional elements orfeatures.

FIG. 9 is a flowchart of a method 900 for generating a projection imageof a three-dimensional sample. The method 900 includes obtaining a setof images of the sample, wherein each image of the set of imagescorresponds to a respective focal plane within the sample (910). Themethod 900 additionally includes applying a filter to each image of theset of images to determine a respective depth value for each pixel of adepth map, wherein a given depth value represents a depth, within thesample, at which the contents of the sample can be imaged in-focus(920). The method 900 further includes determining an image value foreach pixel of an output image based on the depth value of acorresponding pixel of the depth map (930). The method 900 could includeadditional elements or features.

FIG. 10 is a flowchart of a method 1000 for segmenting an image of asample. The method 1000 includes obtaining an image of the sample(1010). The method 1000 additionally includes obtaining a depth map ofcontents of the sample (1020). The method 1000 further includesgenerating a first segmentation map of the sample based on the image(1030). The method 1000 further includes, based on the depth map,generating a second segmentation map of the sample by further dividingat least one region of the first segmentation map (1040). The method1000 could include additional elements or features.

VIII. CONCLUSION

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context indicates otherwise. The illustrativeembodiments described in the detailed description, figures, and claimsare not meant to be limiting. Other embodiments can be utilized, andother changes can be made, without departing from the scope of thesubject matter presented herein. It will be readily understood that theaspects of the present disclosure, as generally described herein, andillustrated in the figures, can be arranged, substituted, combined,separated, and designed in a wide variety of different configurations,all of which are explicitly contemplated herein.

With respect to any or all of the message flow diagrams, scenarios, andflowcharts in the figures and as discussed herein, each step, blockand/or communication may represent a processing of information and/or atransmission of information in accordance with example embodiments.Alternative embodiments are included within the scope of these exampleembodiments. In these alternative embodiments, for example, functionsdescribed as steps, blocks, transmissions, communications, requests,responses, and/or messages may be executed out of order from that shownor discussed, including in substantially concurrent or in reverse order,depending on the functionality involved. Further, more or fewer steps,blocks and/or functions may be used with any of the message flowdiagrams, scenarios, and flow charts discussed herein, and these messageflow diagrams, scenarios, and flow charts may be combined with oneanother, in part or in whole.

A step or block that represents a processing of information maycorrespond to circuitry that can be configured to perform the specificlogical functions of a herein-described method or technique.Alternatively or additionally, a step or block that represents aprocessing of information may correspond to a module, a segment, or aportion of program code (including related data). The program code mayinclude one or more instructions executable by a processor forimplementing specific logical functions or actions in the method ortechnique. The program code and/or related data may be stored on anytype of computer-readable medium, such as a storage device, including adisk drive, a hard drive, or other storage media.

The computer-readable medium may also include non-transitorycomputer-readable media such as computer-readable media that stores datafor short periods of time like register memory, processor cache, and/orrandom access memory (RAM). The computer-readable media may also includenon-transitory computer-readable media that stores program code and/ordata for longer periods of time, such as secondary or persistent longterm storage, like read only memory (ROM), optical or magnetic disks,and/or compact-disc read only memory (CD-ROM), for example. Thecomputer-readable media may also be any other volatile or non-volatilestorage systems. A computer-readable medium may be considered acomputer-readable storage medium, for example, or a tangible storagedevice.

Moreover, a step or block that represents one or more informationtransmissions may correspond to information transmissions betweensoftware and/or hardware modules in the same physical device. However,other information transmissions may be between software modules and/orhardware modules in different physical devices.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

We claim:
 1. A method for generating a projection image of athree-dimensional sample, the method comprising: obtaining a set ofimages of the sample, wherein each image of the set of imagescorresponds to a respective focal plane within the sample; applying afilter to each image of the set of images to determine a respectivedepth value for each pixel of an output image of the sample, wherein agiven depth value represents a depth, within the sample, at which thecontents of the sample can be imaged in-focus; and determining an imagevalue for each pixel of the output image based on the depth value of thepixel of the output image, wherein determining an image value for aparticular pixel of the output image comprises: (i) identifying an imageof the set of images that corresponds to the depth value of theparticular pixel; and (ii) determining the image value for theparticular pixel based on a pixel, of the identified image, having alocation within the identified image that corresponds to the particularpixel.
 2. The method of claim 1, wherein the filter is a texture filter,and wherein applying the filter to a particular image of the set ofimages comprises, for a particular pixel of the particular image,determining at least one of a standard deviation, an entropy, or anumerical range of pixels of the particular image that are in aneighborhood of the particular pixel.
 3. The method of claim 1, furthercomprising generating a depth map for the sample based on the depthvalues.
 4. The method of claim 1, further comprising: generating a firstsegmentation map of the sample based on the output image; and based onthe determined depth values, generating a second segmentation map of thesample by further dividing at least one segment of the firstsegmentation map.
 5. The method of claim 4, wherein further dividing atleast one segment of the first segmentation map comprises: selecting,from the determined depth values, a set of depth values that correspondto a particular region of the first segmentation map; identifying atleast two clusters within the selected set of depth values; and furtherdividing the particular region based on the identified at least twoclusters.
 6. The method of claim 1, further comprising: spatiallypre-processing the determined depth values, wherein determining theimage value for each pixel of the output image based on the depth valueof the pixel of the output image comprises determining the image valuefor each pixel of the output image based on the spatially pre-processeddepth value of the pixel of the output image.
 7. The method of claim 1,wherein the set of images is a set of brightfield images of the sample.8. The method of claim 1, wherein the set of images is a set offluorescent images of the sample.
 9. The method of claim 1, wherein thesample contains at least one three-dimensional cultured multicellularstructure.
 10. The method of claim 9, wherein the at least onethree-dimensional cultured multicellular structure includes at least oneof an organoid embedded in extracellular matrix or a tumor spheroid. 11.The method of claim 10, wherein the at least one three-dimensionalcultured multicellular structure includes an organoid embedded inextracellular matrix, wherein the extracellular matrix has the form of adome of extracellular matrix.
 12. The method of claim 10, wherein the atleast one three-dimensional cultured multicellular structure includes atleast one of a hepatic-cell organoid, a pancreatic-cell organoid, or anintestinal-cell organoid.
 13. The method of claim 1, wherein determiningthe image value for the particular pixel of the output imageadditionally comprises: (iii) identifying two or more additional imagesof the set of images that correspond to depth values within aneighborhood of the depth value of the particular pixel, and whereindetermining the image value for the particular pixel based on the pixel,of the identified image, having a location within the identified imagethat corresponds to the particular pixel comprises performing apixel-wise operation on pixels of the identified image and the two ormore additional images that have locations within their respectiveidentified images that correspond to the particular pixel.
 14. A methodfor generating a projection image of a three-dimensional sample, themethod comprising: obtaining a set of images of the sample, wherein eachimage of the set of images corresponds to a respective focal planewithin the sample; applying a filter to each image of the set of imagesto determine a respective depth value for each pixel of a depth map,wherein the depth value represents a depth, within the sample, at whichcontents of the sample can be imaged in-focus; and determining an imagevalue for each pixel of an output image based on the depth value of acorresponding pixel of the depth map.
 15. The method of claim 14,wherein the filter is a texture filter, and wherein applying the filterto a particular image of the set of images comprises, for a particularpixel of the particular image, determining at least one of a standarddeviation, an entropy, or a numerical range of pixels of the particularimage that are in a neighborhood of the particular pixel.
 16. The methodof claim 14, further comprising: generating a first segmentation map ofthe sample based on the output image; and based on the determined depthvalues, generating a second segmentation map of the sample by furtherdividing at least one region of the first segmentation map.
 17. Themethod of claim 16, wherein further dividing at least one region of thefirst segmentation map comprises: selecting, from the determined depthvalues, a set of depth values that correspond to a particular region ofthe first segmentation map; identifying at least two clusters within theselected set of depth values; and further dividing the particular regionbased on the identified at least two clusters.
 18. The method of claim14, wherein the set of images is a set of brightfield images of thesample.
 19. The method of claim 14, wherein the set of images is a setof fluorescent images of the sample.
 20. The method of claim 14, whereinthe sample contains at least one organoid.
 21. A method for segmentingan image of a sample, the method comprising: obtaining an image of thesample; obtaining a depth map of contents of the sample; generating afirst segmentation map of the sample based on the image; and based onthe depth map, generating a second segmentation map of the sample byfurther dividing at least one region of the first segmentation map. 22.The method of claim 21, wherein the depth map comprises a plurality ofdepth values, and wherein further dividing at least one region of thefirst segmentation map comprises: selecting, from the depth values, aset of depth values that correspond to a particular region of the firstsegmentation map; identifying at least two clusters within the selectedset of depth values; and further dividing the particular region based onthe identified at least two clusters.
 23. The method of claim 21,wherein the sample contains at least one organoid. 24-49. (canceled)